Object tracking

ABSTRACT

Embodiments are disclosed herein that relate to the automatic tracking of objects. For example, one disclosed embodiment provides a method of operating a mobile computing device having an image sensor. The method includes acquiring image data, identifying an inanimate moveable object in the image data, determining whether the inanimate moveable object is a tracked object, ate moveable object is a tracked object, then storing information regarding a state of the inanimate moveable object, detecting a trigger to provide a notification of the state of the inanimate moveable object, and providing an output of the notification of the state of the inanimate moveable object.

BACKGROUND

The tracking of the location or other stale of objects may occupy asignificant amount of effort in everyday life. Further, much time may bespent in trying to locate misplaced objects. For example, searching formisplaced car keys, wallets, mobile devices, and the like may causepeople to lose productive time. Likewise, forgetting that the milkcarton in the home refrigerator is almost empty may lead to an extratrip to the store that could have been avoided had the shopperremembered the state of the milk carton. In some instances, such objectsmay be moved, emptied, etc. by a person other than the owner, therebycomplicating the task of tracking.

SUMMARY

Accordingly, various embodiments are disclosed herein that relate to theautomatic tracking of objects. For example, one disclosed embodimentprovides a method of operating a mobile computing device that comprisesan image sensor. The method includes acquiring image data, identifyingan inanimate moveable object in the image data, determining whether theinanimate moveable object is a tracked object, and if the inanimatemoveable object is a tracked object, then storing information regardinga state of the inanimate moveable object. The method further comprisesdetecting a trigger to provide a notification of the state of theinanimate moveable object, and providing an output of the notificationof the state of the inanimate moveable object.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A depicts a user of an embodiment of a see-through display deviceviewing a use environment through the see-through display device.

FIG. 1B shows a perspective of the user of FIG. 1A, and also shows anembodiment of a tracked object alert displayed on the see-throughdisplay device.

FIG. 2 shows another embodiment of a tracked object alert displayed onthe see-through display device.

FIG. 3 shows a block diagram of an embodiment of a use environment for asee-through display device configured to track objects.

FIG. 4 shows a flow diagram depicting an embodiment for tracking objectsvia a see-through display device.

FIG. 5 shows a block diagram depicting an embodiment of a computingdevice.

DETAILED DESCRIPTION

As mentioned above, tracking and remembering the states (e.g. locationor other physical state) of objects may take significant time in effortduring everyday life. Further, failure to accurately track the slates ofobjects may lead to lost time and productivity. Therefore, embodimentsare disclosed herein that relate to automatically tracking the states ofobjects. Briefly, a mobile device, such as a see-through display worn bya user, may comprise image sensors that observe the user's environment.Video data from the image sensors may be processed to detect objects inthe user's environment, and to identify tracked objects in the videodata. State information, such as location and other characteristics, oftracked objects may then be stored. This ma allow for the output of analert regarding the state of a tracked object upon detection of an alerttrigger, such as a user request or a contextual clue. The stateinformation may be stored locally, and/or uploaded to a remote service.Where multiple people utilize object-tracking devices as disclosedherein, each person may become aware of changes to objects made by otherusers via the sharing of object tracking information. In this manner, auser may be able to discover a most recent location of lost keys, may beprovided with a reminder to buy more milk while browsing the dailysection at a grocery store, and/or may track and recall other objectstate information in any suitable manner.

FIGS. 1A and 1B illustrate a non-limiting example use scenario. First,FIG. 1A illustrates a user wearing a head-mounted see-through displaydevice 100 in the form of a pair of glasses, and observing anenvironment (a living room) through the see-through display device 100at a first time t_(i). As described in more detail below, thesee-through display device 100 comprises one or more outwardly facingimage sensors (RGB and/or depth sensors) that acquire video imagery asthe user moves about. The acquired video imagery may be analyzed, inreal time or at a later time, to identify moveable inanimate objectsthat are tracked by the user, as described in more detail below, in FIG.1A, an example embodiment of a tracked moveable inanimate object isillustrated as keys 102 that are located on a table under a pile ofmagazines, but it will be understood that any suitable object may betracked.

The identified object data and/or image data may be stored to providefuture information, such as alerts, to the user regarding the objects.For example, referring to FIG. 1B, at a later time t₂, the keys 102 havebeen covered by a newspaper 104. Thus, in response to a user inputrequesting information about the location of the keys (e.g. a voicecommand detected via a microphone on the see-through display device100), the keys are highlighted 106 on the see-through display device100.

It will be understood that the see-through display device 100 may nottrack state information for all objects that a user observes or eveninteracts with. For example, in FIG. 1A, the environment observed by theuser includes various other moveable inanimate objects than keys 102,such as magazines 108, a flower pot 110, and books 112 on a bookshelf,that the user may not wish to track.

As such, the see-through display device 100 may track the state ofobjects that are considered sufficiently “important,” wherein the term“important” signifies whether sufficient implicit and/or explicitinformation has been received to designate the object as tracked. Thus,before storing any information regarding the observed state of aninanimate moveable object, the see-through display device 100 maydetermine whether the object is a tracked object, and then store thestate information if the object is a tracked object. The see-throughdisplay device 100 also may detect information that changes an“importance score” assigned to objects. As described in more detailbelow, such an importance score may be compared to one or more thresholdimportance scores to determine whether to track object state informationand/or what object state information to track.

An inanimate moveable object may be defined as tracked in any suitablemanner. For example, the status of the object as tracked may bepre-defined (e.g. by a developer), may be user-defined, and/or may beadaptively learned. In embodiments in which some objects are pre-definedas tracked, the pre-definition may be applied to a set of objectsconsidered to be important to most users, such as keys, wallets,financial transaction cards, and the like. Further, users may have theability to remove an object that matches a pre-defined important objectdefinition from the tracked object list, should tracking not be desiredfor that object.

An object may be designated as tracked in any suitable manner. Forexample, the see-through display device 100 may include an objectdesignation mode in which the user holds or otherwise places an objectwithin a field of view of an image sensor on the see-through displaydevice 100, scans an image of the object, and identifies it as tracked.As a more specific example, a user wearing the see-through displaydevice 100 may use a voice command such as “track this object” whileholding an inanimate moveable object (e.g. keys, wallet, credit/debitcards, etc.) in front of the see-through display device 100, therebytriggering the see-through display device to store the object as atracked object.

Further, in some embodiments, a user may be able to place tune and/orlocation constraints on a tracked object designation. For example, auser who is traveling abroad may designate a passport as a trackedobject for a specified date range or until returning to a specifiedlocation (e.g. home), such that the tracking will automatically end whenthe specified date, location, or other condition is met. Additionally, auser may have the option of changing a status of an inanimate moveableobject from tracked to untracked at anytime. It will be understood thatthese methods for the user designation of tracked objects are presentedfor the purpose of example, and that a user may define a tracked objectin any suitable manner.

As mentioned above, in some embodiments the see-through display device100 may adaptively learn which objects to track. This may help a personlocate objects that have not been explicitly defined as tracked objects,yet are of potential importance to the person. The see-through displaydevice 100 may adaptively designate an inanimate moveable object as atracked object based upon any suitable criteria. For example, thesee-through display device 100 may designate an inanimate moveableobject as a tracked object based upon a number, pattern, and/orfrequency of sightings of and/or user interactions with the object, suchthat objects that are seen and/or interacted with more often are morelikely to be considered important.

Further, the see-through display device 100 may recognize specificlocations (e.g. home, office, and other locations), times, and/orsituations (e.g. getting ready to go to work), and determine whether todesignate an inanimate moveable object as a tracked object based upon alocation and/or situation in which the user interacts with the object orin which the object is otherwise seen. For example, objects located at auser's home bookshelf may be more likely to be objects of importancethan objects located on a store shelf. As another example, objects thatare placed down in close proximity to tracked objects may be more likelyto be important than other objects. For example, a user may place keys,wallet, and mobile device down on a table together as a group. As yetanother example, an object that is moved by another person may heconsidered to he potentially important.

Further, in some embodiments, a user may be able to designate a locationand/or a time as a designated important location and/or time. In thismanner, objects that the user interacts with in the designated locationand/or time may be more likely to be important than in times and/orlocations that are not designated as important. As an example, a usermay designate a travel document carrier as an important object for theduration of a vacation. Similarly, objects that a user carries from hometo another location may be more likely to be objects of importance thanobjects originating from the other location. For example, at arestaurant, it may be more desirable to track an object that a userremoves from a coat pocket and places on a table than the user's waterglass.

As yet another example, objects that a user interacts with at certaintimes of the day (e.g. in the morning before going to work) may be morelikely to be objects of importance than objects with which the userinteracts at other times. Further, objects with which a user interactsin temporal proximity to a change of user context (e.g. before leavinghome to go to work) may be likely to be important, as a user mayinteract with keys, wallet, a briefcase, smart phone, and the likebefore leaving home or the office. Objects with which the user interactsat the end of variable length events (e.g. time spent searching for keyseach morning before finding the keys and/or exiting the house) also maybe considered more likely to be important.

An inanimate moveable object may be adaptively designated as tracked inany suitable manner. For example, as mentioned above, an importancescore may be assigned to detected and identified objects in video imagedata based upon locational, temporal, behavioral, and/or other factorsrelating to the interaction of a user of the see-through display devicewith the objects. If the importance score meets or exceeds a thresholdimportance score, then the object may be designated as tracked, and moredetailed information regarding the state of the object (e.g. most recentlocation, historical locations, value of a variable physical property ofthe object etc.) may be stored. Further, in some embodiments, differentthreshold importance scores may be used to define different levels ofimportance. This may allow different types of information to be stored,different types of alerts to be provided, etc. for an object based upondifferent levels of importance.

Patterns of user interactions with objects may be tracked in anysuitable manner. For example, user behaviors may be classified basedupon contextual information regarding the time, location, type ofactivity being performed, and/or any other suitable contextualinformation. Example user behaviors include eating meals, showering,looking for lost objects, driving to work, and the like. By tracking andstoring information related to such interactions over time, patterns ofinteraction with detected objects may be observed. For example, it maybe observed via the see-through display device 100 that every afternoonat five o'clock a user sets down keys, and looks for the keys everymorning at eight o'clock.

Alerts also may be generated based upon observed user patterns. Forexample, once a behavioral pattern related to user interactions withkeys is recognized, a reminder regarding a location of the keys may bedisplayed automatically each morning at eight o'clock AM. Such patternsalso may be used to formulate guesses as to where a lost object may beif the object has been moved since its location was last observed andstored via the see-through display device. An example of a displayednotice offering suggestions as to where to search for a lost object isshown in FIG. 2 in the form of a list 200 of locations at which to lookfor the object. FIG. 2 also shows a notice 202 regarding where theobject was last seen. It will be understood that a notice regardingwhere to look for a lost object may take any form. For example, such anotice may take the form of graphical instructions that lead the user tothe possible location(s), such as arrows and/or lines to follow,highlighting pointing out the suggested location to look, etc.

In some embodiments, a single view of a location may be generated basedupon inputs of video data from multiple users, for example, where eachuser is uploading video data to a cloud-based service. This may allowobjects seen in one user's video stream to be correlated with objectsseen in another user's video stream, and therefore may allow objects tobe tracked even when they are moved by other users. Further, objectmetadata may be shared across users so that multiple users can uploadtracked object data instead of video data (e.g. such that multipledevices may perform local recognition for the same object).

As mentioned above, in addition to location, other states of objects maybe understood and monitored. As non-limiting examples, a level of milkin a milk carton may be monitored. Likewise, a list of contents in arefrigerator may be monitored over time, and the absence of a item thatis usually present may be noted. Based upon these observed states,alerts may be generated by a contextual trigger. For example, in thecase of an observed empty milk carton, a visual overlay, anauto-generated shopping list, and/or a verbal reminder may be generatedas the user walks by the milk section in the grocery store. Thecontextual trigger of being in the milk section of the grocery store maybe understood in any suitable manner, including but not limited to byannotation or tagging of a particular type/brand of product by thegrocery store or milk producer, by matching the appearance, size, and/orother physical and/or contextual characteristics of observed objects inthe milk section with the characteristics of the tracked milk object,etc. Further, the fullness state of milk cartons may be adaptivelylearned as a tracked object state over time as the pattern of usershopping compared to the milk carton fullness state is observed overtime (i.e. user consistently buys more milk when state of milk carton isempty or near empty).

FIG. 3 shows a block diagram of a use environment 300 for the automatedtracking of inanimate moveable objects. The use environment 300 shows anarbitrary number N of object tracking devices, illustrated as objecttracking device 1 302 and object tracking device N 304, in communicationwith a remote object tracking service 306 via a network 307, such as acomputer network. It will be understood that the object tracking devicesmay take any suitable form, including but not limited to the see-throughdisplay device 100 of FIG. 1. It will further be appreciated that someembodiments may omit a remote object tracking service.

The object tracking device 302 comprises one or more sensors 308. Thesensors 308 may include image sensors, such as as two-dimensional imagesensor(s) 310 and/or depth sensor(s) 312, which collect video data of auser's local environment. Any suitable type and number oftwo-dimensional image sensors 310 and/or depth sensors 312 may beincluded. Examples of suitable two-dimensional image sensors include RGBand grayscale sensors. Examples of suitable depth sensors includetime-of-flight sensors, structured light sensors, and stereo depthsensors. It will be understood that any other suitable sensors may beincluded. Examples include, but are not limited to, microphones, globalpositioning system (GPS) sensors, motion sensors, inward-facing imagesensors to detect eye motion, etc.

Image data collected via the image sensors is provided to an objectrecognition and tracking module 314, which identifies objects imaged bythe image sensors, and detects state information regarding the objects.The object recognition and tracking module 314 further may providealerts to one or more outputs 316, such as a display 318 (e.g. asee-through display, or any other suitable display) and/or one or morespeakers 320, based upon object state, as described in more detailbelow. The object recognition and tracking module 314 may identifyobjects in any suitable manner, including but not limited to viaclassification functions pre-trained by a developer, and/orclassification functions trained by a user to recognize user-specifiedobjects, wherein the classification functions compare and/or fit theobserved objects to object models 315. The object recognition andtracking module 314 may then output object identifications correspondingto the objects identified in the image data to an object informationstore 322. Further, where new objects are detected (i.e. objects thathave not been previously detected), the object recognition and trackingmodule 314 may assign identifications to the objects.

The object information store 322 may be configured to store trackedobject data 324, including but not limited to identity and stateinformation for tracked objects. The object information store 322 alsomay store untracked object data 326 in embodiments that adaptively learnuser patterns and automatically designate objects as tracked, whereinthe term “untracked” indicates that state information is not stored forsuch objects. The untracked object data 326 is illustrated as storinginformation on a plurality of untracked objects, illustrated as object 1328 and object N 332. Any suitable untracked object data may be stored.For example, importance score information 330 may be stored for eachuntracked object, wherein the importance score information 330 may beassigned to a recognized object based upon user interactions with theobject and then used to determine whether to track the state of anobject. When the importance score exceeds a threshold importance score,the object may be designed as a tracked object, and state informationmay be stored for the object.

The tracked object data 324 may include any suitable data stored for anysuitable number and type of tracked objects. In FIG. 3, an plurality oftracked objects is indicated as tracked object 1 336 and tracked objectN 337. Various state data 338 is also illustrated for tracked object 1.For example, location information 340, including but not limited to amost recent location and/or past locations, may be stored. Such locationinformation also may comprise information regarding past userinteractions with an object. This may allow for the analysis of userpatterns to help locate lost objects, for example. Any suitable locationinformation may be stored. Examples include information regarding aphysical location of a current use environment (e.g. GPS coordinates)and/or contextual location data (e.g. location relative to othertracked/recognized objects).

Further, information regarding a value of a variable property 342 of theobject may be stored. For example, as described above, an absence of aparticular food item in a refrigerator, a level of milk in a milkcarton, and/or any other suitable value related to a variable propertyof an object may be stored. It will be understood that these particulartypes of state information are described for the purpose of example, andthat any other suitable state information may be stored.

Other information than state information also may be stored for anobject. For example, in embodiments that adaptively learn and designateobjects as tracked, importance score data 344 may be stored for trackedobjects to determine whether to maintain a designation of an object astracked. For example, if a user stops interacting with an object that isnot lost, over time the score for that object may decrease, and theobject may cease to be tracked. Further, alert conditions 346 also maybe stored, wherein the alert conditions define when alerts are to betriggered for specified objects, as well as a nature of an alert to beprovided.

Additionally, contextual data 350 may be stored as object information tohelp in the determination of an importance score for an object. Forexample, contextual data 350 may define locations 352, times 354, and/orother contextual information as “important” for the purpose of assigningan importance score to an object. Examples of locations 352 may includeactual physical locations, contextual rules regarding proximity to othertracked objects, and other such location data. Examples of times 354 mayinclude clock/calendar times, and/or contextual rules regarding timeintervals between observed object interactions and other events, such aschanges in location (e.g. time between interacting with an object andthen leaving the house). The contextual data 350 further may comprisecontextual rules regarding how to apply location, time, and othercontextual information in determining importance scores for objects.

In some embodiments, the analysis of image data for objects may beperformed at a later time, rather than in real time. As such, the objectinformation store 322 may include image data 355 that has not beenprocessed by the object recognition and tracking module 314. In suchembodiments, image analysis may be performed when a user requestsinformation about an object to locate the object in the image data bylocating relevant image data in the stored image data 355. To assistwith locating relevant image data, various metadata may be stored withthe image data, such as a location (e.g. GPS data) and/or a time atwhich the image data was acquired.

In some embodiments, the object tracking and storage may be performedlocally on the object tracking device 302. In other embodiments, asmentioned above, the object tracking device 302 may communicate withremote object tracking service 306 via a network. This may allow objectdata and image data to be shared between users. The remote objecttracking service 306 may perform any of the object recognition,tracking, and alert generation functions described above with regard tothe object tracking device 302. Further, information received from usersmay be stored in a user information store 356, which is illustrated asstoring information for a plurality of users represented by user 1 358and user N 359. Any suitable information may be stored, including butnot limited to object data 360 (e.g. tracked and untracked objectinformation), image data 362 (e.g. point cloud depth data and/ortwo-dimensional image data), and contextual data 364 (e.g.places/times/other contexts used for determining whether to trackobjects with which a user interacts). Further, the user data also maycomprise information regarding trusted other users 366 with whom objectdata, image data, and/or other information may be shared. For example, auser may wish to access the image data of other family members to aid inthe detection of lost objects. As a more specific example, if a user'sspouse moves the user's keys from a table into a drawer, the spouse'sobject and/or image data may be searched along with the user's objectand/or image data to help locate the keys.

FIG. 4 shows a flow diagram depicting an embodiment of a method 400 fortracking objects via a see-through display device. Method 400 comprises,at 402, receiving image data of a background scene viewable through asee-through display system from an image sensor, wherein the term“background scene” denotes a field of view of a real-world scene locatedbehind the see-through display relative to a user. Any suitable imagedata may be acquired, including but not limited to two-dimensional videodata 404 (RGB and/or grayscale) and depth data 406.

Method 400 next comprises, at 408, identifying an inanimate moveableobject in the image data, either in real time or at a later time uponreceipt of a trigger (e.g. user request to locate an object, contextualtrigger, etc.), and at 410, determining whether the detected inanimatemoveable object is a tracked object. As described above, the inanimatemoveable object may be identified in any suitable manner, including butnot limited to via classification methods that compare the moveableobject to object models.

Likewise, as described above, an inanimate moveable object may bedefined as tracked in any suitable manner. For example, as indicated at412, a user may request (via voice, gesture, and/or other suitable userinput) that an object be designated as tracked, and scan an image of theobject. Additionally, some objects may be designated as important by adeveloper and/or manufacturer of the see-through display device.

Further, as indicated at 414, an inanimate moveable object may bedesignated as tracked if a score (“importance score”) that is assignedbased upon user interactions with the object meets a thresholdimportance score. Any suitable factor or combination of factors may beused to determine such a score. For example, an importance score may bedetermined at least partially based upon a number, frequency, and/orpattern of user interactions with the object over time, as indicated at416. Further, locational information also may be used in determining animportance score, as indicated at 418. Such locational information mayinclude, but is not limited to, whether the object is in a location thatis designated as important, as indicated at 420, and/or whether theobject is in sufficiently close proximity to another tracked object, asindicated at 422.

Further, an importance score also may be assigned based at leastpartially upon a time at which a user interaction occurs, as indicatedat 424. The time may comprise a designated important time 426, atemporal proximity between the user interaction with the object and achange in user context or other notable event, and/or any other suitabletemporal factors. It will be understood that a location and/or a timemay be designated as important in any suitable manner, including but notlimited to by a user, by a developer and/or manufacturer, via adaptivelearning, etc. It also will be understood that an importance score maybe assigned in any other suitable manner. For example, in someembodiments, a user may assign an importance score to an object. In suchembodiments, the user-assigned score may be assigned permanently, or maybe an initial value that is later varied based upon user interactionswith the object. Likewise, in other embodiments, an application maychoose an importance score based upon any other suitable factors thanlocation and/or time.

If the inanimate moveable object is determined to be a tracked objectthen method 400 comprises, at 430, storing information regarding thestate of the object. Stored state information may include, but is notlimited to, location information and/or information regarding a value ofa variable physical property of the object. Further, he stateinformation may be stored locally, as indicated at 432, and/or sent to aremote service for remote storage, as indicated at 434.

Method 400 next comprises, at 436, detecting a trigger to provide aoutput of notification of a state of a tracked object. Any suitabletrigger may be used. For example, as indicated at 438, the trigger maycomprise a user input requesting information regarding the state of theobject, such as a request to locate a lost object. Further, as indicatedat 440, the trigger may comprise a detected value of a variable physicalproperty of the object meeting a threshold condition. As yet anotherexample, as indicated at 442, the trigger may comprise a detected time,location, and/or other object that is contextually associated with atracked object. For example, detecting that a user is walking through adairy section of a grocery store may trigger the output of anotification of a reminder to buy milk if an absence/low level of milkhas previously been detected in the user's refrigerator. Likewise, wherea user is at a location other than home, detecting a trigger maycomprise detecting that a user has moved a predetermined distance from apersonal item. As a more specific example, a trigger may be detectedwhere a user sets down a wallet in a restaurant and then walks away fromthe table or outside of the building. It will be understood that thetriggers described above are presented for the purpose of example, andare not intended to be limiting in any manner.

Upon detecting the trigger, method 400 comprises, at 444, providing anoutput of a notification of the slate of the object. Any suitable outputmay be provided, including audio 446 and/or visual outputs 448. Examplesof suitable visual outputs include, but are not limited to, augmentedreality imagery using a see-through display device, as described abovewith regard to FIG. 1. Further, information regarding the state may beobtained from local storage, or from a remote service. Where theinformation is obtained from a remote service, the information may beobtained from the user's account, another trusted user's account, asdescribed above, or from any other suitable source.

The notification may comprise any suitable information. For example, asindicated at 450, the notification may inform a user of a most recentlystored location of the object. As more specific examples, thenotification may highlight a physical location of an object in abackground scene, and/or may comprise directions (arrows, lines, text,etc.) that lead the user to a most recent location. A notification alsomay comprise an alert that a user has left the object behind in a publicplace. Further, the notification may comprise information regarding adetected value of a variable physical property of the object. Forexample, in the case where a milk carton meets a threshold state ofemptiness, the notification may comprise a reminder to buy more milk,and/or an option to perform an online purchase of more milk.

Further, in some embodiments, a notification may include an image of theobject being sought. This may help to locate an object in cases wheresimilar looking variants of the object exist. For example, if a user isseeking a set of keys and a display device has information on four setsof keys that look similar, the display device may show images of thefour sets of keys to allow the user to select the set being sought.Further, an image of a current setting of the object may be displayed.For example, if a set of keys being sought are currently on anightstand, the display device may display an image of the keys on thenightstand to show the user the actual location of the keys.

In this manner, the states of objects may be tracked automatically forusers, thereby facilitating the location of lost objects and/or thetracking of any other suitable object state. In some embodiments, theabove described methods and processes may be tied to a computing systemincluding one or more computers. In particular, the methods andprocesses described herein may be implemented as a computer application,computer service, computer API, computer library, and/or other computerprogram product.

FIG. 5 schematically shows a nonlimiting computing system 500 that mayperform one or more of the above described methods and processes.Computing system 500 is shown in simplified form. It is to be understoodthat virtually any computer architecture may be used without departingfrom the scope of this disclosure. In different embodiments, computingsystem 500 may take the form of a mainframe computer, server computer,desktop computer, laptop computer, tablet computer, home entertainmentcomputer, network computing device, mobile computing device, mobilecommunication device, see-through display device, near-eye displaydevice, gaining device, etc., including but not limited to thesee-through displays and other computing devices described herein withreference to FIGS. 1-4.

Computing system 500 includes a logic subsystem 502 and a data-holdingsubsystem 504. Computing system 500 may optionally include a displaysubsystem 508, communication subsystem 506, and/or other components notshown in FIG. 5. Computing system 500 may also optionally include userinput devices such as keyboards, mice, game controllers, cameras,microphones, and/or touch screens, for example.

Logic subsystem 502 may include one or more physical devices configuredto execute one or more instructions. For example, the logic subsystemmay be configured to execute one or more instructions that are part ofone or more applications, services, programs, routines, libraries,objects, components, data structures, or other logical constructs. Suchinstructions may be implemented to perform a task, implement a datatype, transform the state of one or more devices, or otherwise arrive ata desired result.

The logic subsystem may include one or more processors that areconfigured to execute software instructions. Additionally oralternatively, the logic subsystem may include one or more hardware orfirmware logic machines configured to execute hardware or firmwareinstructions. Processors of the logic subsystem may be single core ormulticore, and the programs executed thereon may be configured forparallel or distributed processing. The logic subsystem may optionallyinclude individual components that are distributed throughout two ormore devices, which may be remotely located and/or configured forcoordinated processing. One or more aspects of the logic subsystem maybe virtualized and executed by remotely accessible networked computingdevices configured in a cloud computing configuration.

Data-holding subsystem 504 may include one or more physical,non-transitory, devices configured to hold data and/or instructionsexecutable by the logic subsystem to implement the herein describedmethods and processes. When such methods and processes are implemented,the state of data-holding subsystem 504 may be transformed (e.g., tohold different data).

Data-holding subsystem 504 may include removable media and/or built-indevices. Data-holding subsystem 504 may include optical memory devices(e.g., CD, DVD HD-DVD, Blu-Ray Disc, etc.), semiconductor memory devices(e.g., RAM, EPROM, EEPROM, etc.) and/or magnetic memory devices (e.g.,hard disk drive, floppy disk drive, tape drive, MRAM, etc.), amongothers. Data-holding subsystem 504 may include devices with one or moreof the following characteristics: volatile, nonvolatile, dynamic,static, read/write, read-only, random access, sequential access,location addressable, file addressable, and content addressable. In someembodiment, logic subsystem 502 and data-holding subsystem 504 may beintegrated into one or more common devices, such as an applicationspecific integrated circuit or a system on a chip.

FIG. 5 also shows an aspect of the data-holding subsystem in the form ofremovable computer-readable storage media 510, which may be used tostore and/or transfer data and/or instructions executable to implementthe herein described methods and processes. Removable computer-readablestorage media 510 may take the form of CDs, DVDs, HD-DVDs, Blu-RayDiscs, EEPROMs, and/or floppy disks, among others.

It is to be appreciated that data-holding, subsystem 504 includes one ormore physical, non-transitory devices. In contrast, in some embodimentsaspects of the instructions described herein may be propagated in atransitory fashion by a pure signal (e.g., an electromagnetic signal, anoptical signal, etc.) that is not held by a physical device for at leasta finite duration. Furthermore, data and/or other forms of informationpertaining to the present disclosure may be propagated by a pure signal.

The terms “module” and “program” may be used to describe an aspect ofcomputing system 500 that is implemented to perform one or moreparticular functions. In some cases, such a module and/or program may beinstantiated via logic subsystem 502 executing instructions held bydata-holding subsystem 504. It is to be understood that differentmodules and/or programs may be instantiated from the same application,service, code block, object, library, routine, API, function, etc.Likewise, the same module and/or program may be instantiated bydifferent applications, services, code blocks, objects, routines, APIs,functions, etc. The terms “module” and “program” are meant to encompassindividual or groups of executable files, data files, libraries,drivers, scripts, database records, etc.

It is to be appreciated that a “service”, as used herein, may be anapplication program executable across multiple user sessions andavailable to one or more system components, programs, and/or otherservices. In some implementations, a service may run on a serverresponsive to a request from a client.

Display subsystem 508 may be used to present a visual representation ofdata held by data-holding subsystem 504. As the herein described methodsand processes change the data held by the data-holding subsystem, andthus transform the state of the data-holding subsystem, the state ofdisplay subsystem 508 may likewise be transformed to visually representchanges in the underlying data. Display subsystem 508 may include one ormore display devices utilizing virtually any type of technology,including but not limited to see-through display technologies. Suchdisplay devices may be combined with logic subsystem 502 and/ordata-holding subsystem 504 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, communication subsystem 506 may be configured tocommunicatively couple computing system 500 with one or more othercomputing devices. Communication subsystem 506 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As nonlimiting examples, the communicationsubsystem may be configured for communication via a wireless telephonenetwork, a wireless local area network, a wired local area network, awireless wide area network, a wired wide area network, etc. In someembodiments, the communication subsystem may allow computing system 500to send and/or receive messages to and/or from other devices via anetwork such as the Internet.

It is to be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated may beperformed in the sequence illustrated, in other sequences, in parallel,or in some cases omitted. Likewise, the order of the above-describedprocesses may be changed.

The subject matter of the present disclosure includes all novel andnonobvious combinations and subcombinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A method of operating a mobile computing device, the computing devicecomprising an image sensor, the method comprising: acquiring image data;identifying an inanimate moveable object in the image data; determiningwhether the inanimate moveable object is a tracked object; if theinanimate moveable object is a tracked object, then storing informationregarding a state of the inanimate moveable object; detecting a triggerto provide a notification of the state of the inanimate moveable object;and providing an output of the notification of the state of theinanimate moveable object.
 2. The method of claim 1, further comprisingdetermining whether the inanimate moveable object is a tracked objectbefore storing information regarding the state of the inanimate moveableobject.
 3. The method of claim 2, further comprising receiving a userinput assigning the inanimate moveable object as a tracked object and/orassigning a user-selected importance score to the inanimate moveableobject.
 4. The method of claim 2, further comprising assigning theinanimate moveable object an importance score based upon userinteractions with the inanimate moveable object, and designating theinanimate moveable object as a tracked object based upon the importancescore meeting a threshold importance score.
 5. The method of claim 4,wherein the importance score is assigned based at least partially upon anumber of user interactions with the inanimate moveable object.
 6. Themethod of claim 4, wherein the importance score is assigned based eastpartially upon one or more of a location of the inanimate moveableobject and a time at which a user interaction with the inanimatemoveable object occurs.
 7. The method of claim 6, wherein the importancescore is assigned at least partially upon one or more of the location ofthe inanimate moveable object being designated as an important location,a proximity of the inanimate moveable object to other tracked objects,and a temporal proximity between a user interaction with the inanimatemoveable object and a change in user context.
 8. The method of claim 1,wherein the state of the inanimate moveable object comprises a locationof the inanimate moveable object, and wherein the output of thenotification comprises an output of a notification of the location ofthe object.
 9. The method of claim 1, wherein the state of the inanimatemoveable object comprises a value of a variable physical property of theinanimate moveable object, and wherein detecting the trigger comprisesdetecting that the value of the variable physical property meets athreshold condition.
 10. The method of claim 1, wherein detecting thetrigger comprises detecting one or more of a time, a location, and anobject contextually associated with the inanimate moveable object. 11.The method of claim 1, wherein detecting the trigger comprises receivinga user input requesting information regarding the state.
 12. The methodof claim 1, further comprising storing the information regarding thestate by sending the information to a remote service via a network. 13.The method of claim 1, wherein the image data is acquired from one ormore of an image sensor and a remote service.
 14. A mobile computingdevice, comprising: a sensor subsystem comprising an image sensor; adisplay subsystem; a logic subsystem configured to execute instructions;and a data-holding subsystem comprising instructions stored thereon thatare executable by the logic subsystem to: receive image data from theimage sensor; detect via the image data a plurality of user interactionswith a moveable inanimate object over; assign an importance score to theinanimate moveable object based upon the plurality of user interactionswith the inanimate moveable object; if the importance score meets athreshold importance score, then store information regarding a state ofthe inanimate moveable object; detect a trigger to provide anotification of the state of the inanimate moveable object; and providean output of the notification of the state of the inanimate moveableobject.
 15. The mobile computing device of claim 14, wherein the stateof the inanimate moveable object comprises a value of a variablephysical property of the inanimate moveable object, and wherein theinstructions are executable to detect the trigger by detecting that thevalue of the variable physical property meets a threshold condition. 16.The mobile computing device of claim 14, wherein the instructions areexecutable to detect the trigger by detecting a user input requestinginformation regarding the state of the inanimate moveable object.
 17. Ahead-mounted display device, comprising a sensor subsystem comprising animage sensor; a display subsystem comprising a see-through display; alogic subsystem configured to execute instructions; and a data-holdingsubsystem comprising instructions stored thereon that are executable bythe logic subsystem to: receive a user input from a user requesting animage scan of a inanimate moveable object and also requesting a state ofthe inanimate moveable object to be tracked; in response, acquire animage of the inanimate moveable object; store information regarding alocation at which the inanimate moveable object is set down by the user;receive a user input requesting information regarding the location ofthe inanimate moveable object; determine that the inanimate moveableobject is not currently at the updated location, and in response,provide a notification regarding potential other locations of theinanimate moveable object.
 18. The head-mounted display device of claim17, further comprising determining the potential other locations basedupon past user behaviors.
 19. The head-mounted display device of claim17, wherein the information regarding the location of the inanimatemoveable object comprises image data.
 20. The head-mounted displaydevice of claim 17, further comprising, upon detecting the trigger,obtaining a most recent state of the inanimate moveable object from aremote service.